Skip to content

feat(feedback): thumbs up/down on answers (MCP + chat) with an answer_log table - #1162

Merged
northdpole merged 299 commits into
stagingfrom
feat/answer-feedback
Nov 1, 2026
Merged

northdpole merged 299 commits into
stagingfrom
feat/answer-feedback

Conversation

@northdpole

Copy link
Copy Markdown
Collaborator

Adds thumbs up/down on answers so staging can collect labelled data.

  • answer_log table (+ migration d1e2f3a4b5c6, which joins the two existing heads) and recording of ratings.
  • MCP: ask returns an answer_id; new rate_answer tool.
  • Chat: completion returns answer_id; POST /rest/v1/answer_feedback; GET /admin/feedback. Chat answers are logged only when a deployment opts in (ANSWER_LOG_CHAT=1).
  • Chat UI: thumbs under answers, storage notice when answers are logged, and the "not stored" privacy paragraph is hidden when answers are stored.
  • Test: alembic_single_head_test fails when the migration chain has more than one head.

Stacked on #1158 (order: #1142, #1155, #1159, #1158, this). Needs the migration run and ANSWER_LOG_CHAT=1 set on staging. ask now writes one row per question, so this should be deployed after measurement runs.

Tested: answer_feedback_test, alembic_single_head_test, mcp_ask_test pass; Black clean. Frontend tests run in CI.

🤖 Generated with Claude Code

northdpole and others added 30 commits October 15, 2026 18:43
Hide the showcase when CRE_ALLOW_IMPORT is off, matching the header
nav and API kill switch. Failed target deletes now show the API error.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
openRun now checks res.ok and does not populate the mapping editor
from error JSON. Successful target add/start/delete clears a stale
error banner.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Skip blank standard names so impact no longer calls get_nodes(""),
return 400 JSON instead of a 500 HTML page on graph failures, and
show impact warnings in the admin review UI.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Drop SQLite file-path probing for the agent index. Config help and
admin status now expect a Postgres URL and ignore other values.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Return JSON error bodies instead of HTML abort pages, ignore bad
repos.yaml on seed, and cover blank/duplicate/disabled target and
mapping edge cases in API and Admin UI tests.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Save writes the packaged harvester yaml. Start one-off uses a temp
copy so custom yaml does not overwrite it. Import review source is a
custom name or repos.yaml:<sha256-12>. GitHub org expand fills
repositories (skipping forks, archived, and existing owner/repo).

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Allowlist GitHub owner logins, quote the API path, cap yaml at 256KiB
before safe_load, and write packaged repos.yaml atomically.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
The 256KiB parse limit rejected a full GitHub org dump. Keep a
size cap, but large enough for up to 1000 repository entries.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Config is a sync write of github.com/org/ (or a single repo URL).
The indexer lists org repos at harvest time and classifies each as
OpenCRE or OWASP-agent material. Add org no longer calls GitHub.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Drop the dedicated Admin tab. Resources lists a virtual owasp-agent
row with status in spec; Start now runs the same harvest as oie_repo.
The built-in id is reserved, skipped on yaml seed, and cannot be
removed unless a leftover DB row exists.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Save, Add org, and one-off ingest probe api.github.com for each sources
URL before writing or starting harvest. Missing, private, or timed-out
orgs/repos return 400 with the canonical URL. The indexer still expands
reachable orgs later.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Admin home lists running jobs and links through to pipeline logs.
Config edits targets.yaml (cron, add-org, add-target form) plus every
.env.example env var with secrets redacted. Pipeline New can harvest
the packaged yaml or a one-off repository. Graph management reviews
per-link approve/deny/relink for each run. MyOpenCRE stays a separate
route.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Add org now writes packaged repos.yaml after the GitHub probe, matching
Add target. Graph management notes that link decisions stay on the staged
run until CRE-edge apply lands. Failed applies no longer look complete.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Remove the separate agent DB env var. Admin status and config read
DATABASE_URL / DEV_DATABASE_URL (same Postgres as CRE + pgvector).

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Admin invoked python /abs/scripts/run_oie_pipeline.py, so sys.path[0]
was scripts/ and `import application` failed. Other threads set
PYTHONPATH= at the repo root. Put the repo on sys.path in the script
and in the admin subprocess env.

IndexStore now defaults to DATABASE_URL / DEV_DATABASE_URL instead of
tmp/owasp_agent.sqlite. OWASP_AGENT_DB remains a test override.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
SQLAlchemy 2.1 maps postgresql:// to psycopg v3; pin psycopg2 via
postgresql+psycopg2://. Admin invoke_oie_cli uses the app Postgres URL
and refuses sqlite. `make admin-local` starts cre-postgres when the URL
is down, migrates, create_all, and launches Flask without CRE_CACHE_FILE.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Production and admin/agent now share main_db_env_keys (DATABASE_URL
before PROD_DATABASE_URL). Rewrite postgresql+psycopg:// to
+psycopg2. Gate local create_all behind ADMIN_LOCAL_CREATE_ALL.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Live defaults sync repos and persist to Postgres; auto-skip B/C without
LLM keys; async API with wait=true for sync probes. Treat harvester
degraded as non-fatal and surface chunks/files in stage detail.
Pass --max-repos through the CLI so admin caps stay effective.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Empty locale stubs in Top10 aborted the whole repo harvest; skip and
continue so golden-set ingest can finish. Admin agent status now counts
all public tables and reports MAX(fetched_at) as last_sync.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Map only Module C safety-path degraded as non-fatal so A/B degraded
halts before B/C again. Reject max_repos < 1. Always skip B/C unless
the client opts in so LLM keys cannot launch expensive stages from New.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Admin /rest JSON errors no longer collapse to HTML "Resource Not found".
List backfills missing staged_change_set rows. Graph UI shows Accept
feedback, disables dead actions, and explains empty CRE changesets on
OIE harvest runs (ops live in harvest_input).

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Orchestrator JSON now includes engine, flags, graph_path, and visited.
Skip stage details explain caller flags; harvester detail notes HEAD
checkpoints when chunks=0. Admin records oie_trace and richer started JSON.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
OIE harvest fills harvest_input but not the explorer graph; run
cre.py --upstream_sync before Flask unless the cre/node tables
already have data (SKIP_UPSTREAM_SYNC / FORCE_UPSTREAM_SYNC overrides).

Co-authored-by: Spyros <northdpole@users.noreply.github.com>
* trim/1158:
  docs(env): OWASP_CX_LONGCTX off and OWASP_CX_LONGCTX_CHARS back to the code default
  fix(oie-scheduler): embed_filed_nodes defaults off, document OIE_HARVEST_EVENTS and OIE_EMBED_FILED_NODES
  fix(owasp-agent): admin sync and scheduler stage apply the opt-in meta-embedding push
  chore(owasp-agent): type embed_sync and its test
  chore: drop generated frontend bundle (rebuild at deploy)
  refactor(owasp-agent): move the meta-embedding push out of sync_all into embed_sync
…ed Google email

auth_callback matched LOGIN_ALLOWED_DOMAINS with email.endswith(domain), so
attacker@evilowasp.org passed for owasp.org, and never read email_verified.
Reuse the roles domain rule (exact domain or subdomain, same as the MCP
sign-in) and refuse unverified addresses, named admins included.
Existing callback tests now carry email_verified.
* trim/1127:
  test(owasp-agent): sync and concepts must not import application.database
* trim/1137:
  test(owasp-agent): sync and concepts must not import application.database
* trim/1136:
  test(owasp-agent): sync and concepts must not import application.database
* trim/1142:
  fix(admin): CRE embedding guard only treats a missing table as empty and counts real vectors
  test(owasp-agent): sync and concepts must not import application.database
* trim/1155:
  fix(admin): CRE embedding guard only treats a missing table as empty and counts real vectors
  test(owasp-agent): sync and concepts must not import application.database
* trim/1159:
  fix(admin): CRE embedding guard only treats a missing table as empty and counts real vectors
  test(owasp-agent): sync and concepts must not import application.database
* trim/1158:
  fix(admin): CRE embedding guard only treats a missing table as empty and counts real vectors
  test(owasp-agent): sync and concepts must not import application.database
…-style domains, require email_verified is True
* trim/1127:
  test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests
* trim/1137:
  test(owasp-agent): exercise the meta-embedding push through injected embed_fn instead of patching embed_sync
  test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests

# Conflicts:
#	application/tests/oie_scheduler/runner_test.py
#	application/utils/oie_scheduler/jobs.py
* trim/1136:
  test(oie-scheduler): restore the environment with a plain helper in the new opt-in assertions
  test(owasp-agent): exercise the meta-embedding push through injected embed_fn instead of patching embed_sync
  test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests

# Conflicts:
#	application/tests/admin_panel_test.py
#	application/utils/admin_panel/service.py
* trim/1142:
  docs(staging): use placeholder tester addresses; test: restore the environment with a plain helper in the #1142 tests
  test(oie-scheduler): restore the environment with a plain helper in the new opt-in assertions
  test(owasp-agent): exercise the meta-embedding push through injected embed_fn instead of patching embed_sync
  test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests
* trim/1155:
  docs(staging): use placeholder tester addresses; test: restore the environment with a plain helper in the #1142 tests
  test(oie-scheduler): restore the environment with a plain helper in the new opt-in assertions
  test(owasp-agent): exercise the meta-embedding push through injected embed_fn instead of patching embed_sync
  test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests
* trim/1159:
  test(owasp-agent): restore the environment with a plain helper in index_routing_test
  docs(staging): use placeholder tester addresses; test: restore the environment with a plain helper in the #1142 tests
  test(oie-scheduler): restore the environment with a plain helper in the new opt-in assertions
  test(owasp-agent): exercise the meta-embedding push through injected embed_fn instead of patching embed_sync
  test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests
* trim/1158:
  test(owasp-agent): restore the environment with a plain helper in index_routing_test
  docs(staging): use placeholder tester addresses; test: restore the environment with a plain helper in the #1142 tests
  test(oie-scheduler): restore the environment with a plain helper in the new opt-in assertions
  test(owasp-agent): exercise the meta-embedding push through injected embed_fn instead of patching embed_sync
  test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests
… callback instead of patching them; plain env helper; placeholder addresses in .env.example
northdpole added a commit that referenced this pull request Nov 1, 2026
* trim/1162: (27 commits)
  docs(env): document CHAT_BLOCK_UNVERIFIED (default on, changes public chat answers)
  trim(#1162): drop prune_noise_chunks (unguarded DELETE on embeddings), validate authority weights before use, document ANSWER_LOG_*, no host paths in sync_check_staging
  trim(#1158): strip_injection no longer truncates legitimate questions; correct default-off claims
  trim(#1159): drop cambridge->boston alias, dead locals, test env/tempfile leaks
  trim(#1155): drop laptop-local edge probe scripts, make node metadata seeding opt-in, fix empty-key blurb match
  trim(#1142): admin upstream-sync-and-ingest no longer replaces existing CRE embeddings
  trim(#1142): no credentialed CORS without an allowlist, OIE workers opt-in in import-all, guard levers-matrix queue reset
  trim(#1137): drop brevity-bot skills and stale handover doc, restore README/dev setup, loopback guard for run_admin_local.sh
  trim(#1127): drop adversarial asker/batch scripts and canned CRE follow-ups, set source_channel in raw embedding writers
  trim(#1120): fix merge-conflict markers, guard full pipeline, drop dead experiments
  style: black-format llm_safety_guard.py
  chore(oie): trim #1118 for the staging rollout
  chore(oie): trim #1117 for the staging rollout
  chore(oie): trim #1116 and guard destructive eval scripts
  chore(harvester): trim #1115 for the staging rollout
  chore(oie): trim #1088 for the staging rollout
  fix(auth): harden web sign-in found by adversarial review
  fix(auth): /auth/user needs any role, not admin
  fix(auth): carry the PKCE code_verifier from login to callback
  fix(auth): build a fresh OAuth flow per login instead of a shared singleton
  ...
@northdpole
northdpole changed the base branch from feat/ingest-comprehension to staging November 1, 2026 22:51
@northdpole
northdpole merged commit 89463ee into staging Nov 1, 2026
10 checks passed
@northdpole
northdpole deleted the feat/answer-feedback branch November 1, 2026 22:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant